Optimizing delay-sensitive network-based communications with latency guidance

ABSTRACT

Devices, computer-readable media, and methods for selecting a type of transmission for an immersive visual stream based upon a latency estimate. For instance, a processing system including at least one processor may obtain a latency estimate for an immersive visual stream, determine whether the latency estimate exceeds a latency threshold for selecting a type of transmission for the immersive visual stream, and select the type of transmission for the immersive visual stream from among a field of view restricted type of transmission and a field of view plus out of scene type of transmission based upon the determining. When the latency estimate is determined to not exceed the latency threshold, the field of view restricted type of transmission is selected. When the latency estimate is determined to exceed the latency threshold, the field of view plus out of scene type of transmission is selected.

This application is a continuation of U.S. patent application Ser. No.16/285,066, filed on Feb. 25, 2019, now U.S. Pat. No. 10,659,190, whichis herein incorporated by reference in its entirety.

The present disclosure relates generally to optimizing delivery ofdelay-sensitive network-based communications, such as immersive visualstreams, and more particularly to devices, non-transitorycomputer-readable media, and methods for selecting a type of packet lossprotection for a network-based communication based upon a latencyestimate, and to devices, non-transitory computer-readable media, andmethods for selecting a type of transmission for an immersive visualstream based upon a latency estimate.

BRIEF DESCRIPTION OF THE DRAWINGS

The teachings of the present disclosure can be readily understood byconsidering the following detailed description in conjunction with theaccompanying drawings, in which:

FIG. 1 illustrates an example network related to the present disclosure;

FIG. 2 illustrates an example system in accordance with the presentdisclosure;

FIG. 3 illustrates an example frame of a visual stream in accordancewith the present disclosure;

FIG. 4 illustrates a flowchart of an example method for selecting a typeof packet loss protection for a network-based communication based upon alatency estimate, in accordance with the present disclosure;

FIG. 5 illustrates a flowchart of an example method for selecting a typeof transmission for an immersive visual stream based upon a latencyestimate, in accordance with the present disclosure; and

FIG. 6 illustrates a high level block diagram of a computing devicespecifically programmed to perform the steps, functions, blocks and/oroperations described herein.

To facilitate understanding, identical reference numerals have beenused, where possible, to designate identical elements that are common tothe figures.

DETAILED DESCRIPTION

In one example, the present disclosure describes a device,computer-readable medium, and method for selecting a type of packet lossprotection for a network-based communication based upon a latencyestimate. For instance, a processing system including at least oneprocessor may obtain a latency estimate for a network-basedcommunication, determine whether the latency estimate exceeds a latencythreshold for selecting a type of packet loss protection, and select thetype of packet loss protection for the network-based communication fromamong a first type of packet loss protection and a second type of packetloss protection based upon the determining. When the latency estimate isdetermined to not exceed the latency threshold, the first type of packetloss protection is selected. When the latency estimate is determined toexceed the threshold, the second type of packet loss protection isselected.

In another example, the present disclosure describes a device,computer-readable medium, and method for selecting a type oftransmission for an immersive visual stream based upon a latencyestimate. For instance, a processing system including at least oneprocessor may obtain a latency estimate for an immersive visual stream,determine whether the latency estimate exceeds a latency threshold forselecting a type of transmission for the immersive visual stream, andselect the type of transmission for the immersive visual stream fromamong a field of view restricted type of transmission and a field ofview plus out of scene type of transmission based upon the determining.When the latency estimate is determined to not exceed the latencythreshold, the field of view restricted type of transmission isselected. When the latency estimate is determined to exceed thethreshold, the field of view plus out of scene type of transmission isselected.

Delay sensitive network-based communications may include videoconferencing, cloud-based video gaming, immersive augmented reality (AR)or virtual reality (VR) media, video streaming, e.g., high definitiontwo-dimensional (2D) video, three-dimensional (3D) video, and/orvolumetric video, and many more. For such delay sensitive applications,it is beneficial to account for and to minimize, if possible, theend-to-end delay to less than a couple of hundreds of milliseconds,otherwise, the user experience may be degraded. For example with respectto a streaming video application, the end-to-end latency may includevideo encoding latency, transmission latency, decoding latency, etc. Inother words, the end-to-end latency may include a summation of thedelays associated with the processes involved in: 1) the obtaining andencoding of the video by a server, e.g., a video server, a conferencingserver, or a gaming server; 2) the transport of the encoded video overone or more networks; and 3) the decoding and displaying of the video ona client device. Some of these processes can be controlled and managedfrom the server side and also from the client side, e.g., the processesrunning on the server and/or client devices can be properly monitoredand controlled. However, there are more uncertainties as to the networkcomponents forming the underlying network resources utilized in theforwarding of the video between the server side and the client side. Forexample, user demands may rise and fall without advance warning and theperformance of the underlying network resources is closely tied to suchdemands. In addition, the Internet may be involved in the transport ofthe video between the server and the client device. Since the Internetis a best effort network, network dynamics, including bandwidthfluctuation, latency fluctuation, packet losses and jitter are to beexpected.

To address network packet loss, an error-correcting code (ECC), e.g.,forward error correction (FEC), may be applied across n applicationlayer source video packets to generate k application layer redundant FECpackets. If the total number of received packets m, including bothsource packets and redundant packets, is larger than the number oforiginal source video packets n, the lost packets can be fullyrecovered. It may (or may not) introduce certain encoding delay, but itin general does not introduce much transmission delay. Thus theadvantage of application layer FEC is that it does not introduce muchadditional delay and the additional delay is largely configurable anddeterministic. Furthermore, the packet loss rate that it can handle isalso configurable and deterministic. The disadvantage of applicationlayer FEC is that it may waste bandwidth and its performance depends onthe packet loss rate, as (1) if there are less packet losses thanredundant FEC packets, certain redundant FEC packets will be wasted; andmore importantly (2) if the number of lost packets is higher than thenumber of redundant FEC packets, application layer FEC does not work,and all redundant packets are wasted.

Similarly, a retransmission scheme may be applied to address networkpacket loss. In this case, if a packet is lost, a client may requestthat a server retransmit a lost packet. It does not introduce muchoverhead and uses bandwidth more efficiently. However, if the end-to-endlatency is too long, the retransmitted packet may miss the decoding anddisplay deadline, inducing degraded visual quality of experience (QoE).For instance, the client device 210A may maintain a shallow buffer forreceiving and assembling packets, where the buffer depth may be set suchthat media (e.g., video frames, audio frames, etc.) is played-outwithout perceptible delay to a user.

In one example of the present disclosure, latency guidance is used toadapt delay sensitive network-based applications to select differenttypes of packet loss protection schemes. The adaptation is not limitedto the two above-listed schemes and other schemes can be evaluated andselected, such as an automatic repeat request (ARQ) scheme, a hybridautomatic repeat request (HARQ) scheme, a redundant stream scheme, andso forth. However, for illustrative purposes, the present examples aredescribed primarily in connection with an error-correcting code, e.g.,FEC, and retransmission schemes. It is again noted that applicationlayer FEC is latency friendly but it is less bandwidth efficient; whileretransmission is bandwidth efficient but it is less latency friendly.In one example, if latency guidance predicts the near-term latency willbe low (e.g., below a threshold, such as under 200 ms for VR or 3D videostreaming), a retransmission-based scheme may be selected. Otherwise,application layer FEC may be selected.

In addition, for immersive visual streaming, such as 3D video, 360video, volumetric video, VR or AR media, e.g., for 3D gaming, remotesurgery, or other applications, and so forth, in one example of thepresent disclosure, latency guidance is used to adapt delay sensitivenetwork-based applications to select different types of transmission. Toillustrate, immersive video, e.g., 360 degree video, three dimensional(3D) video, volumetric video, and so forth may be captured by multiplevideo cameras, video stitched, compressed, and streamed to user devicesfor decoding and display. To provide an immersive watching experience, aresolution of 4K or higher after video stitching may be considered aminimum acceptable frame resolution, while 8K resolution or higher maybe preferred. The according video bitrate can be as high as tens tohundreds of Mbps. However, wireless spectrum and network bandwidth maybe limited (and costly), especially with predicted millions of usersstreaming 360/3D videos over the Internet simultaneously.

In one example, an immersive video frame may include a field-of-view(FoV), e.g., the viewing area in a stitched image, and a non-FoV, e.g.,regions which are available but not currently visible to a user orpresented via a display. One approach is to transmit the entire frame(e.g., the entire image of the frame), including both of the FoV and therest of the image. In this way, no matter how a viewer changes the FoV,the viewer can immediately view this FoV, as the entire image istransmitted. However, since for a certain time the viewer only views theFoV, the bandwidth used to deliver data relating to non-FoV regions iswasted. Another approach is that only data relating to the FoV istransmitted (broadly “FoV restricted”). This approach may effectivelyreduce the network bandwidth utilization. However, FoV prediction may bedifficult to achieve. In addition, without accurate FOV prediction,buffered data at a client/user endpoint device for the predicted FoV iswasted. More importantly, if the visual data of the actual FoV cannot bereceived in a very short time (in the scale of several ms), the imagewill be frozen and QoE will be degraded. Another approach involvestwo-tier video quality transmission, where a full frame/image of lowerquality is always transmitted, while data for rendering a higher visualquality for the FoV is transmitted based on FoV prediction. In this way,even if the predicted FoV is wrong, there is still a lower qualityvisual data for any actual FoV.

In one example of the present disclosure, a network latency predictionis used to select a more aggressive or less aggressive transmission. Forinstance, if the predicted network latency is lower than a certainthreshold (e.g., 20 ms), a more aggressive video transmission scheduleis selected. For example, in this scenario, only visual data for thepredicted FoV area is transmitted. When the FoV prediction is wrong, theclient device may retrieve the visual data for the actual FoV area fromthe server. Since latency is low (e.g., less than 20 ms), it is expectedthat these bytes can be received before any QoE degradation can beobserved.

On the other hand, if the predicted network latency is higher than acertain threshold (e.g., 20 ms), a more conservative video transmissionschedule is selected. For example, in this scenario, an image of 360degree video, volumetric video, or the like can be encoded intotwo-tiers (or layers with layered video), where the FoV area is encodedand transmitted with high quality video while non-FoV area is encodedand transmitted with low quality video. As it is predicted that latencyis high, low quality video of the non-FoV area is used to provide basicvisual quality when the actual FoV area has not been received.

Thus, examples of the present disclosure, including both for selecting atype of packet loss protection for a network-based communication basedupon a latency estimate and for selecting a type of transmission for animmersive visual stream based upon a latency estimate may achievenetwork bandwidth savings when possible in connection with low latencyscenarios, while avoiding QoE degradation when latency is high byadapting to different packet loss protection schemes or types oftransmission. These and other aspects of the present disclosure aredescribed in greater detail below in connection with the examples ofFIGS. 1-6.

To better understand the present disclosure, FIG. 1 illustrates anexample network 100, related to the present disclosure. As shown in FIG.1, the network 100 connects mobile devices 157A, 157B, 159, 167A, 167B,and 169, and home network devices such as home gateway 161, set-topboxes (STBs) 162A and 162B, television (TV) 163A and TV 163B, home phone164, router 165, personal computer (PC) 166, and so forth, with oneanother and with various other devices via a core network 110, awireless access network 150 (e.g., a cellular network), an accessnetwork 120, other networks 140, content distribution network (CDN) 170,and/or the Internet in general. For instance, connections between corenetwork 110, access network 120, home network 160, CDN 170, wirelessaccess network 150 and other networks 140 may comprise the Internet ingeneral, internal links under the control of single telecommunicationservice provider network, links between peer networks, and so forth.

In one example, wireless access network 150 comprises a radio accessnetwork implementing such technologies as: Global System for MobileCommunication (GSM), e.g., a Base Station Subsystem (BSS), or IS-95, aUniversal Mobile Telecommunications System (UMTS) network employingWideband Code Division Multiple Access (WCDMA), or a CDMA3000 network,among others. In other words, wireless access network 150 may comprisean access network in accordance with any “second generation” (2G),“third generation” (3G), “fourth generation” (4G), Long Term Evolution(LTE) or any other yet to be developed future wireless/cellular networktechnology. While the present disclosure is not limited to anyparticular type of wireless access network, in the illustrative example,wireless access network 150 is shown as a UMTS terrestrial radio accessnetwork (UTRAN) subsystem. Thus, elements 152 and 153 may each comprisea Node B or evolved Node B (eNodeB). In one example, wireless accessnetwork 150 may be controlled and/or operated by a same entity as corenetwork 110.

In one example, each of mobile devices 157A, 157B, 159, 167A, 167B, and169 may comprise any subscriber/customer endpoint device configured forwireless communication such as a laptop computer, a Wi-Fi device, aPersonal Digital Assistant (PDA), a mobile phone, a smartphone, an emaildevice, a computing tablet, a messaging device, a wearable computingdevice (e.g., smart glasses, augmented reality glasses, or a headset), awireless speaker and/or smart speaker, a digital voice assistant device,and the like. In one example, any one or more of mobile devices 157A,157B, 159, 167A, 167B, and 169 may have both cellular and non-cellularaccess capabilities and may further have wired communication andnetworking capabilities.

As illustrated in FIG. 1, network 100 includes a core network 110. Inone example, core network 110 may combine core network components of acellular network with components of a triple play service network; wheretriple play services include telephone services, Internet services andtelevision services to subscribers. For example, core network 110 mayfunctionally comprise a fixed mobile convergence (FMC) network, e.g., anIP Multimedia Subsystem (IMS) network. In addition, core network 110 mayfunctionally comprise a telephony network, e.g., an InternetProtocol/Multi-Protocol Label Switching (IP/MPLS) backbone networkutilizing Session Initiation Protocol (SIP) for circuit-switched andVoice over Internet Protocol (VoIP) telephony services. Core network 110may also further comprise a broadcast television network, e.g., atraditional cable provider network or an Internet Protocol Television(IPTV) network, as well as an Internet Service Provider (ISP) network.The network elements 111A-111D may serve as gateway servers or edgerouters to interconnect the core network 110 with other networks 140,wireless access network 150, access network 120, and so forth. As shownin FIG. 1, core network 110 may also include a plurality of television(TV) servers 112, and a plurality of application servers 114. For easeof illustration, various additional elements of core network 110 areomitted from FIG. 1.

With respect to television service provider functions, core network 110may include one or more television servers 112 for the delivery oftelevision content, e.g., a broadcast server, a cable head-end, and soforth. For example, core network 110 may comprise a video super huboffice, a video hub office and/or a service office/central office. Inthis regard, television servers 112 may include content server(s) tostore scheduled television broadcast content for a number of televisionchannels, video-on-demand programming, local programming content, and soforth. Alternatively, or in addition, content providers may streamvarious contents to the core network 110 for distribution to varioussubscribers, e.g., for live content, such as news programming, sportingevents, and the like. Television servers 112 may also includeadvertising server(s) to store a number of advertisements that can beselected for presentation to viewers, e.g., in the home network 160 andat other downstream viewing locations. For example, advertisers mayupload various advertising content to the core network 110 to bedistributed to various viewers. Television servers 112 may also includeinteractive TV/video-on-demand (VOD) server(s), as described in greaterdetail below.

In one example, the access network 120 may comprise a Digital SubscriberLine (DSL) network, a broadband cable access network, a Local AreaNetwork (LAN), a cellular or wireless access network, a 3^(rd) partynetwork, and the like. For example, the operator of core network 110 mayprovide a cable television service, an IPTV service, or any other typeof television service to subscribers via access network 120. In thisregard, access network 120 may include a node 122, e.g., a mini-fibernode (MFN), a video-ready access device (VRAD) or the like. However, inanother example, node 122 may be omitted, e.g., forfiber-to-the-premises (FTTP) installations. Access network 120 may alsotransmit and receive communications between home network 160 and corenetwork 110 relating to voice telephone calls, communications with webservers via other networks 140, content distribution network (CDN) 170and/or the Internet in general, and so forth. In another example, accessnetwork 120 may be operated by a different entity from core network 110,e.g., an Internet service provider (ISP) network.

Alternatively, or in addition, the network 100 may provide televisionand/or other data services to home network 160 via satellite broadcast.For instance, ground station 130 may receive television content fromtelevision servers 112 for uplink transmission to satellite 135.Accordingly, satellite 135 may receive television content from groundstation 130 and may broadcast the television content to satellitereceiver 139, e.g., a satellite link terrestrial antenna (includingsatellite dishes and antennas for downlink communications, or for bothdownlink and uplink communications), as well as to satellite receiversof other subscribers within a coverage area of satellite 135. In oneexample, satellite 135 may be controlled and/or operated by a samenetwork service provider as the core network 110. In another example,satellite 135 may be controlled and/or operated by a different entityand may carry television broadcast signals on behalf of the core network110.

As illustrated in FIG. 1, core network 110 may include variousapplication servers 114. For instance, application servers 114 may beimplemented to provide certain functions or features, e.g., aServing-Call Session Control Function (S-CSCF), a Proxy-Call SessionControl Function (P-CSCF), or an Interrogating-Call Session ControlFunction (I-CSCF), one or more billing servers for billing one or moreservices, including cellular data and telephony services, wire-linephone services, Internet access services, and television services.Application servers 114 may also include a Home Subscriber Server/HomeLocation Register (HSS/HLR) for tracking cellular subscriber devicelocation and other functions. An HSS refers to a network elementresiding in the control plane of an IMS network that acts as a centralrepository of all customer specific authorizations, service profiles,preferences, etc. Application servers 114 may also include an IMS mediaserver (MS) for handling and terminating media streams to provideservices such as announcements, bridges, and Interactive Voice Response(IVR) messages for VoIP and cellular service applications. The MS mayalso interact with customers for media session management. In addition,application servers 114 may also include a presence server, e.g., fordetecting a presence of a user. For example, the presence server maydetermine the physical location of a user or whether the user is“present” for the purpose of a subscribed service, e.g., online for achatting service and the like.

In one example, application servers 114 may include a latency guidanceserver. For instance, a latency guidance server may comprise a computingsystem or server, such as computing system 600 depicted in FIG. 6, andmay be configured to provide one or more operations or functions forselecting a type of packet loss protection for a network-basedcommunication based upon a latency estimate, for selecting a type oftransmission for an immersive visual stream based upon a latencyestimate, or for both. It should be noted that as used herein, the terms“configure,” and “reconfigure” may refer to programming or loading aprocessing system with computer-readable/computer-executableinstructions, code, and/or programs, e.g., in a distributed ornon-distributed memory, which when executed by a processor, orprocessors, of the processing system within a same device or withindistributed devices, may cause the processing system to perform variousfunctions. Such terms may also encompass providing variables, datavalues, tables, objects, or other data structures or the like which maycause a processing system executing computer-readable instructions,code, and/or programs to function differently depending upon the valuesof the variables or other data structures that are provided. As referredto herein a “processing system” may comprise a computing deviceincluding one or more processors, or cores (e.g., as illustrated in FIG.6 and discussed below) or multiple computing devices collectivelyconfigured to perform various steps, functions, and/or operations inaccordance with the present disclosure. It should be noted that theforegoing are only several examples of the types of relevant applicationservers 114 that may be included in core network 110 for storinginformation relevant to providing various services to subscribers.

In accordance with the present disclosure, other networks 140 andservers 149 may comprise networks and devices of various mediaproviders. For example, servers 149 may store and provide media content,such as video data, audio data, gaming data, and so forth to variousclient devices, such as one or more of mobile devices 157A, 157B, 159,167A, 167B, 169, and/or PC 166, via other networks 140, core network110, access network 120, wireless access network 150, CDN 140, and soforth. In addition, any one or more of servers 149 may comprise acomputing system or server, such as computing system 600 depicted inFIG. 6, and may be configured to provide one or more operations orfunctions for selecting a type of packet loss protection for anetwork-based communication based upon a latency estimate, for selectinga type of transmission for an immersive visual stream based upon alatency estimate, or for both.

In one example, home network 160 may include a home gateway 161, whichreceives data/communications associated with different types of media,e.g., television, phone, and Internet, and separates thesecommunications for the appropriate devices. The data/communications maybe received via access network 120 and/or via satellite receiver 139,for instance. In one example, television data is forwarded to set-topboxes (STBs)/digital video recorders (DVRs) 162A and 162B to be decoded,recorded, and/or forwarded to television (TV) 163A and TV 163B forpresentation. Similarly, telephone data is sent to and received fromhome phone 164; Internet communications are sent to and received fromrouter 165, which may be capable of both wired and/or wirelesscommunication. In turn, router 165 receives data from and sends data tothe appropriate devices, e.g., personal computer (PC) 166, mobiledevices 167A, 167B, and 169, and so forth. In one example, router 165may further communicate with TV (broadly a display) 163A and/or 163B,e.g., where one or both of the televisions is a smart TV. In oneexample, router 165 may comprise a wired Ethernet router and/or anInstitute for Electrical and Electronics Engineers (IEEE) 802.11 (Wi-Fi)router, and may communicate with respective devices in home network 160via wired and/or wireless connections.

In one example, one or both of the STB/DVR 162A and STB/DVR 162B maycomprise a computing system or server, such as computing system 600depicted in FIG. 6, and may be configured to provide one or moreoperations or functions for selecting a type of packet loss protectionfor a network-based communication based upon a latency estimate. Amongother functions, STB/DVR 162A and STB/DVR 162B may comprise videoplayers capable of playing video programs in formats such as MovingPicture Expert Group (MPEG) .mpeg files, .mov files, .mp4 files, 0.3gpfiles, .f4f files, .m3u8 files, or the like. Although STB/DVR 162A andSTB/DVR 162B are illustrated and described as integrated devices withboth STB and DVR functions, in other, further, and different examples,STB/DVR 162A and/or STB/DVR 162B may comprise separate STB and DVRdevices.

In addition, one or more of mobile devices 157A, 157B, 159, 167A, 167B,169, and/or PC 166 may also comprise a computing system, such ascomputing system 600 depicted in FIG. 6, and may be configured toprovide one or more operations or functions for selecting a type ofpacket loss protection for a network-based communication based upon alatency estimate, for selecting a type of transmission for an immersivevisual stream based upon a latency estimate, or for both. A flowchart ofan example method of selecting a type of packet loss protection for anetwork-based communication based upon a latency estimate is illustratedin FIG. 4 and described in greater detail below. A flowchart of anexample method of selecting a type of transmission for an immersivevisual stream based upon a latency estimate is illustrated in FIG. 5 anddescribed in greater detail below.

Network 100 may also include a content distribution network (CDN) 170.In one example, CDN 170 may be operated by a different entity from corenetwork 110. In another example, CDN 170 may be operated by a sameentity as core network 110, e.g., a telecommunication service provider.In one example, the CDN 170 may comprise a collection of cache serversdistributed across a large geographical area and organized in a tierstructure. The first tier may comprise a group of servers that accesscontent web servers (origin servers) to pull content into the CDN 170,referred to as ingest servers, e.g., ingest server 172. The content mayinclude video programs, content of various webpages, electronicdocuments, video games, etc. A last tier may comprise cache serverswhich deliver content to end user, referred to as edge caches, or edgeservers, e.g., edge server 174. For ease of illustration, a singleingest server 172 and a single edge server 174 are shown in FIG. 1. Inbetween the ingest server 172 and edge server 174, there may be severallayers of servers (omitted from the illustrations), referred to as themiddle tier. In one example, the edge server 174 may be multi-tenant,serving multiple content providers, such as core network 110, contentproviders associated with server(s) 149 in other network(s) 140, and soforth. In addition, in one example, edge server 174 may comprise acomputing system or server, such as computing system 600 depicted inFIG. 6, and may be configured to provide one or more operations orfunctions for selecting a type of packet loss protection for anetwork-based communication based upon a latency estimate, for selectinga type of transmission for an immersive visual stream based upon alatency estimate, or for both.

As mentioned above, TV servers 112 in core network 110 may also includeone or more interactive TV/video-on-demand (VOD) servers. In oneexample, an interactive TV/VOD server may comprise a computing system orserver, such as computing system 600 depicted in FIG. 6, and may beconfigured to provide one or more operations or functions for selectinga type of packet loss protection for a network-based communication basedupon a latency estimate, for selecting a type of transmission for animmersive visual stream based upon a latency estimate, or for both.Among other things, an interactive TV/VOD server may function as aserver for STB/DVR 162A and/or STB/DVR 162B, one or more of mobiledevices 157A, 157B, 159, 167A, 167B, 169, and/or PC 166 operating as aclient for requesting and receiving videos or other immersive visualstreams (e.g., 360 video, 3D video, volumetric video, etc.) as describedherein.

Further details regarding the functions that may be implemented by edgeserver 174, STBs/DVRs 162A and 162B, mobile devices 157A, 157B, 159,167A, 167B, and 169, and/or PC 166 are discussed in greater detail belowin connection with the examples of FIGS. 2-5. In addition, it should berealized that the network 100 may be implemented in a different formthan that which is illustrated in FIG. 1, or may be expanded byincluding additional endpoint devices, access networks, networkelements, application servers, etc. without altering the scope of thepresent disclosure. For example, core network 110 is not limited to anIMS network. Wireless access network 150 is not limited to a UMTS/UTRANconfiguration. Similarly, the present disclosure is not limited to anIP/MPLS network for VoIP telephony services, or any particular type ofbroadcast television network for providing television services, and soforth.

FIG. 2 illustrates an example system 200 according to the presentdisclosure. As illustrated in FIG. 2, the system 200 includes Internet230, and networks 220A and 220B. For illustrative purposes, the Internet230 may comprise any one or more interconnected networks, such astransport networks, core networks, metro network, etc., of one or moretelecommunication service providers. Internet 230 may also include oneor more content distribution networks (CDNs), public or private cloudcomputing infrastructure, and so forth. In the example of FIG. 2, accessnetworks 220A and 220B may comprise broadband optical and/or cableaccess networks, Local Area Networks (LANs), wireless access networks(e.g., an IEEE 802.11/Wi-Fi network and the like), cellular accessnetworks, Digital Subscriber Line (DSL) networks, public switchedtelephone network (PSTN) access networks, 3^(rd) party networks, and thelike. In one example, the access networks 220A and 220B may comprisedifferent types of access networks or may comprise the same type ofaccess network. The access networks 220A and 220B may be operated bydifferent service providers, the same service provider, or a combinationthereof, or may be operated by entities having core businesses that arenot related to telecommunications services, e.g., corporate,governmental or educational institution LANs, and the like.

Access networks 220A and 220B may transmit and receive communications(broadly, “network-based communications”) between client devices 210Aand 210B, or between server 240 and client device 210A. Client devices210A and 210B may each comprise a personal computer, a laptop computer,a set-top box, a mobile computing device, e.g., a cellular smartphone, awearable computing device (e.g., smart glasses, augmented realityglasses, or a VR and/or AR headset for presenting 360 video, 3D video,volumetric video or similar visual content, including visual streams for3D gaming or other types of delay-sensitive network-based visualapplications, such as for remote surgery, remote teaching, and soforth), a telephone, a wireless speaker and/or smart speaker, a digitalvoice assistant device, and so forth. The client devices 210A and 210Bmay correspond to similar components of FIG. 1, such as STB/DVR 162Aand/or STB/DVR 162B, one or more of mobile devices 157A, 157B, 159,167A, 167B, 169, and/or PC 166. In addition, server 240 may correspondto components of FIG. 1 including TV servers 112, servers 149, edgeserver 174 and so forth, which may comprise servers for deliveringnetwork-based communication, including immersive visual streams, toclient devices.

As illustrated in FIG. 2, the system 200 further includes latencyguidance servers 250A and 250B which may correspond to devices in FIG.1, such as application servers 114, servers 149, etc. In accordance withthe present disclosure, latency guidance servers 250A and 250B maycollect various types of network data from core networks, radio accessnetworks (RANs), and so forth, and may utilize the collected data toestimate/predict the network latency for a network-based communication.In one example, delay over Internet 230 may be assumed to be fairlystatic (e.g., a fixed value or within a defined range) whereas accessnetworks 220A and 220B may have delays which are more variable (inparticular, in examples where access networks 220A and 220B may comprisewireless access networks). Thus, in one example, the delay/latency inaccess networks 220A and 220B may be measured and predicted in order tobe added with the delay/latency over Internet 230 in order to providedelay/latency guidance for a network-based communications via the system200.

To illustrate, core latency (latency of Internet 230) for cellularcommunications is relatively deterministic, and may be related to thedistance between a client device and the network controller. Providedwith the client/user equipment (UE) location, which can be inferred withcell location, and the according network controller, the distance may beknown, as well as the core latency. The more dynamic part of theend-to-end latency is RAN latency. Cellular base stations, such as eNBs,provides rich information and performance indicators (e.g., keyperformance indicators (KPIs)) including physical resource block (PRB)utilization, queue length, reference signal received power (RSRP),reference signal received quality (RSRQ), and so forth that may be usedto predict RAN latency. For instance, in one example, an artificialintelligence/machine learning (AI/ML) algorithm may be applied to thecollected network data to predict near future latency for individualclient devices, network-based communcations, and/or applications.

In one illustrative example, client device 210A may be engaged in anetwork-based communication with server 240, e.g., obtaining a visualstream and/or other media for a cloud-based video game, indicated bysession 260A. In this case, the end-to-end delay, or latency, includesdelay in Internet 230, which as discussed above may be relatively fixed,and the delay in network 220A, e.g., a cellular access network.Continuing with the present example, latency guidance server 250A maycollect network measurements as described above from one or morecomponents of network 220A and may calculate an anticipated latency fornetwork 220A. The delay/latency of network 220A may be added to therelatively fixed delay/latency of Internet 230, and the predictedend-to-end latency may be provided to and/or obtained by client device210A. In one example, the end-to-end latency may be requested by clientdevice 210A (e.g., via a RESTFUL network application programminginterface (API)) and provided by latency guidance server 250A inresponse to the request. Alternatively, or in addition, latency guidanceserver 250A may periodically provide the end-to-end latency predictionsto client device 210A.

Client device 210A may then select a type of packet loss protection fora network-based communication based upon a latency estimate. Forinstance, if the latency prediction is low (e.g., below a threshold,such as under 200 ms for VR or 3D video streaming, under 100 ms, under300 ms, etc.), a retransmission-based scheme may be selected. Otherwise,FEC (e.g., application layer FEC, transport layer FEC, etc.) may beselected. In this regard, it should be noted that various transportlayer protocols may be utilized, such as Transmission Control Protocol(TCP) or Reliable Uniform Datagram Protocol (RUDP) forretransmission-based protection, Uniform Datagram Protocol (UDP) orReal-Time Protocol (RTP) for FEC-based protection, and so forth. In oneexample, client device 210A may instruct or request server 240 to applythe type of packet loss protection scheme that is selected. Forinstance, using reliable transport, client device 210A may send responsepackets indicating that one or more packets in a sequence of thenetwork-based communication from server 240 have been received. As such,client device 210A may include a flag or other informational indicatorsin a header of the response packet indicating the type of packet lossprotection to be utilized, and/or indicating a change in the type ofpacket loss protection. Alternatively, or in addition, client device210A may send a separate packet or other communications (e.g., adatagram, a sequence of packets, etc.) to server 240 indicating when achange in packet loss protection scheme is to be applied.

The foregoing describes an example where the type of packet lossprotection is selected by the client device 210A. However, in anotherexample, the type of packet loss protection may be selected by server240. For instance, server 240 may similarly obtain end-to-end latencyestimates for session 260A from latency guidance server 250A and maysimilarly select the type of packet loss protection based upon the samecriteria as described above, e.g., above or below a threshold, such as200 ms. In such an example, server 240 may provide an indication toclient device 210A of the current type of packet loss protection to useand/or change in the type of packet loss protection, when such a changeis selected. The notification may be in-band, e.g., in one or morepacket headers comprising the network-based communication of session260A, or out-of-band, e.g., via a different network path and/ordifferent session, via one or more packets or other datagrams that donot convey user data (audio, video, etc.), and so forth. In addition, itshould also be noted that in other, further, and different examples,different types of packet loss protection, such as ARQ, HARQ, and soforth, may be similarly considered and selected in accordance with oneor more latency thresholds based upon the end-to-end latency prediction.

As another example, client devices 210A and 210B may be engaged in anetwork-based communication comprising a conference call (e.g., audioonly, or a video call). In such case, the session 260B may include theInternet 230 as well as both networks 220A and 220B. In this example,there may be two different providers of networks 220A and 220B (e.g.,one or both of which may comprise cellular access networks). Inaddition, each of the respective providers may maintain its own latencyguidance server, 250A and 250B, respectively. As such, the end-to-endlatency may involve predictions for both networks 220A and 220B by bothof latency guidance servers 250A and 2506. For example, latency guidanceserver 250A may collect network measurements as described above from oneor more components of network 220A and may calculate an anticipatedlatency for network 220A. Similarly, latency guidance server 250B maycollect network measurements as described above from one or morecomponents of network 220B and may calculate an anticipated latency fornetwork 220B.

In the present example, the predicted delay/latency of network 220A andthe predicted delay/latency of network 220B may be added to therelatively fixed delay/latency of Internet 230 to determine thepredicted end-to-end latency. In one example, client device 210A mayobtain the predicted delay/latency of network 220A and the predicteddelay/latency of network 220B for latency guidance servers 250A and250B, respectively, and may add the relatively fixed delay of Internet230 to calculate the predicted end-to-end delay. In another example, thelatency guidance servers 250A and 250B may exchange latencyestimates/predictions for networks 220A and 220B, and either or both ofthe latency guidance servers 250A and 250B may determine thepredicted/estimated end-to-end delay by adding the predicted latenciesof networks 220A and 220B, and the relatively fixed latency of Internet230. As such, either or both of latency guidance servers 250A and 250Bmay then provide the end-to-end latency estimate to one or both ofclient devices 210A and 210B, e.g., upon request and/or periodically.

In addition, either or both of client devices 210A and 2108 may thenselect a type of packet loss protection in accordance with theend-to-end delay estimate as described above. For instance, the clientdevice that is the caller may be designated to select the type of packetloss protection. In another example, the other client device may bedesignated to select the type of packet loss protection. In anotherexample, the client devices 210A and 2108 may be configured to selectthe type of packet loss protection for inbound packets. For instance,the delays may be not be reciprocal delays, but may be different fordifferent directions. For example, client device 210B may be assignedmore than sufficient downlink bandwidth, but may be allocatedinsufficient uplink bandwidth to support the same data rate as thedownlink. In still another example, either or both of latency guidanceservers 250A and 250B may be included in signaling paths for the session260B such that latency guidance server 250A and/or latency guidanceserver 250B may select a type of packet loss protection in accordancewith the predicted/estimated end-to-end delay, and then may instruct theclient devices 210A and 210B to implement the type of packet lossprotection that is selected. Thus, these and other modifications may beimplemented in additional examples of the present disclosure.

The foregoing examples described in connection with FIG. 2 relate toselecting a type of packet loss protection for a network-basedcommunication based upon a latency estimate. However, examples of thepresent disclosure may alternatively or additionally include selecting atype of transmission for an immersive visual stream based upon a latencyestimate. For instance, the network-based communication of session 260Amay comprise transmission of an immersive visual stream from server 240to client device 210A, such as an AR or VR stream, a 360 video, 3Dvideo, and/or volumetric video stream, etc. Similarly, session 260B maycomprise an AR or VR interaction between client devices 210A and 210B.As described above, client device 210A, client device 210B, and/orserver 240 may obtain end-to-end latency estimates from either or bothof latency guidance servers 250A and 250B or may obtain latencyestimates for networks 220A or 220B, respectively, and may calculate anend-to-end latency therefrom.

In any case, the end-to-end latency estimate may be used to select atype of packet loss protection. However, the end-to-end latency estimatemay alternatively or additionally be used to select a type oftransmission, e.g., where session 260A and/or 260B comprises animmersive visual stream. To illustrate, transmission of an immersivevisual stream may be from server 240 to client device 210A. An exampleframe 300 of an immersive visual stream is illustrated in FIG. 3. Theframe 300 includes a plurality of rectangular tiles 350. As illustratedin FIG. 3, the frame 300 includes a field of view (FoV) area 320. Acentral portion 310 of the FoV area 320 is indicated as labeled.Similarly, a peripheral portion 330 of the FoV area 320 is alsoindicated as labeled. Regions 340 outside of FoV area 320 (e.g.,comprising a plurality of rectangular tiles) are also illustrated inFIG. 3. In one example, a client device, such as client device 210A ofFIG. 2, may determine a FoV based upon one or more sensors, such as agyroscope and compass, a global positioning system (GPS), or the like,via eye and/or gaze tracking, based upon distance estimates to one ormore AR markers via a device camera, and so forth.

As described above, if the predicted end-to-end latency is lower than acertain threshold (e.g., 20 ms, 40 ms, etc.), a more aggressive videotransmission scheme may be selected. For example, in this scenario, onlyvisual data for the predicted FoV area 320 is transmitted by the server240. When the FoV prediction is wrong, the client device 210A mayretrieve the visual data for the actual FoV area from the server 240.Since latency is low (e.g., less than 20 ms, less than 40 ms, etc.), itis expected these bytes can be received before any QoE degradation canbe observed.

On the other hand, if the predicted end-to-end latency is higher than acertain threshold (e.g., 20 ms), a more conservative video transmissionscheme may be selected. For example, in this scenario, an image of 360degree video, volumetric video, or the like can be encoded intotwo-tiers (or layers with layered video), where the FoV area 320 isencoded and/or transmitted by the server 240 with higher visual quality,while non-FoV areas (e.g., regions 340, and the remaining tiles 350outside of FoV area 320, which may be referred to as “out-of-scene”(00S) tiles), are encoded and/or transmitted with lower visual quality.In addition, when the predicted end-to-end latency is high, low qualityvideo of the non-FoV areas is used to provide basic visual quality whenthe actual FoV area 320 has not been received. Although data regardingthe actual FoV area 320 may be requested from the server 240, it isassumed that this information will not reach the client device 210A bythe display deadline.

A similar process for selecting a type of transmission for an immersivevisual stream between client device 210A and client device 210B forsession 260B may be adopted in accordance with the foregoing. Inaddition, it should be noted that variations of the foregoing may alsobe applied. For instance, when selecting a FoV restricted type oftransmission, server 240 may still differentiate visual quality fortiles in the central portion 310 of the FoV area 320 and tiles in theperipheral portion 330 of the FoV area 320 (e.g., higher visual qualityin central portion 310, lower visual quality in the peripheral portion330). Similarly, when selecting for tiered or layered transmission, theserver 240 may implement three or more visual qualities, e.g., a highestvisual quality for the central portion 310, a second highest visualquality for the peripheral portion 330, a third visual quality for theregions 340, a fourth visual quality for remaining tiles 350 in frame300, and so forth. Thus, these and other modifications may beimplemented in additional examples of the present disclosure.

FIG. 4 illustrates a flowchart of a method 400 for selecting a type ofpacket loss protection for a network-based communication based upon alatency estimate, in accordance with the present disclosure. In oneexample, the method 400 is performed by a component of the network 100of FIG. 1, such as by one of application servers 114, servers 149,mobile devices 157A, 157B, 159, 167A, 167B, or 169, home networkdevices, such as STBs 162A and 162B, TV 163A and TV 163B, home phone164, PC 166, and so forth, and/or any one or more components thereof(e.g., a processor, or processors, performing operations stored in andloaded from a memory), or by a plurality of such devices in conjunctionwith one another, and so on. Similarly, the method 400 may be performedby a component of the system 200 of FIG. 2, such as client device 210A,client device 210B, server 240, latency guidance server 250A, or latencyguidance server 250B, and/or any one or more components thereof (e.g., aprocessor, or processors, performing operations stored in and loadedfrom a memory), or by a plurality of such devices in conjunction withone another, and so on. In one example, the steps, functions, oroperations of method 400 may be performed by a computing device orsystem 600, and/or processor 602 as described in connection with FIG. 6below. For instance, the computing device or system 600 may representany one or more components of one or more components of the network 100of FIG. 1 and/or the system 200 of FIG. 2 that is/are configured toperform the steps, functions and/or operations of the method 400.Similarly, in one example, the steps, functions, or operations of method400 may be performed by a processing system comprising one or morecomputing devices collectively configured to perform various steps,functions, and/or operations of the method 400. For instance, multipleinstances of the computing device or processing system 600 maycollectively function as a processing system. For illustrative purposes,the method 400 is described in greater detail below in connection withan example performed by a processing system. The method 400 begins instep 405 and proceeds to step 410.

At step 410, the processing system obtains a latency estimate for anetwork-based communication. The network-based communication maycomprise, for example, at least one of visual data or audio data. Forinstance, video calls, voice calls, conference calls, VR and AR streams,3D video, volumetric video, video gaming streams, and so forth mayinclude visual data (e.g., video, animated graphics, etc.), audio data,or both. The network-based communication may alternatively oradditionally comprise any high data volume and/or delay-sensitive datastreams, such as information feeds for stock trading, premises security,machine learning/artificial intelligence data processing applications,transportation system management, and so forth.

In one example, step 410 may include calculating the latencyestimate/prediction for the network-based communication. For instance,in examples where the network-based communication traverses at least oneradio access network (RAN), the processing system may obtain performanceindicators such as physical PRB utilization, queue length, RSRP, RSRQ,and so forth to predict RAN latency. Similar performance indicators maybe obtained for additional types of access network(s) as well as coreand/or metro network(s), backbone network(s), transport network(s),satellite downlink(s), wireless and or wired local area networks (LANs)wireless wide area network(s) (WWAN(s)), and so forth that may be partof a transmission path for the network-based communication. In exampleswhere there may be more than one network service provider infrastructureinvolved in the conveyance of packets or other protocol data units ofthe network-based communication, step 410 may include obtaining theperformance indicators from the different network service providerinfrastructure according to an arrangement with the respective networkservice provider(s). In one example, an artificial intelligence/machinelearning (AI/ML) algorithm may be applied to the collected performanceindictors to predict near future latency. In addition, in one example,for portions of the network(s) over which the network-basedcommunication is conveyed for which there is no network performancemeasures or latency estimates that are obtainable, a relatively fixedlatency measure may be included in calculating the overall/end-to-endlatency estimate. For instance, Internet latency may be assumed to berelatively fixed, or within a narrow range, and may primarily be afunction of network distance between the participant devices.

At step 420, the processing system determines whether the latencyestimate exceeds a latency threshold for selecting a type of packet lossprotection. The threshold may comprise, for example, 200 ms for VR or 3Dvideo streaming, under 100 ms, under 300 ms, etc.

At step 430, the processing system selects a type of packet lossprotection for the network-based communication from among a first typeof packet loss protection and a second type of packet loss protectionbased upon the determining. In particular, when the latency estimate isdetermined at step 420 to not exceed the latency threshold, the firsttype of packet loss protection is selected, and when the latencyestimate is determined at step 420 to exceed the threshold, the secondtype of packet loss protection is selected. For instance, in oneexample, the first type of pack loss protection may comprise a packetretransmission scheme and the second type of packet loss protection maycomprise an error correcting code, e.g., a forward error correction(FEC) scheme. In other examples, the second type of packet lossprotection may comprise an ARQ or HARQ scheme, a redundant stream scheme(e.g., with different network paths, and in one example, with timeoffset), and so forth.

At optional step 440, the processing system may send an instruction toapply to the network-based communication the type of packet lossprotection that is selected. In one example, the processing systemcomprises at least one device that is a participant of the network-basedcommunication. For instance, the instruction may be sent to a server orendpoint device that is transmitting data packets of the network-basedcommunication. For example, the processing system may comprise thereceiver of the network-based communication or may comprise acentralized network-based processing system that is coordinating thenetwork-based communication (e.g., an application server, a latencyguidance server, etc.). In the case of a centralized network-basedprocessing system, the instruction may be sent to one or bothparticipant devices of the network-based communication (or to multipleparticipant devices, such as for a conference call, a multi-playeronline video game, etc.). For example, the instruction may pertain tothe network-based communication between two of the participant devicesor apply from a server to multiple client devices, e.g., if the clientdevices are in relatively close network proximity and therefore can beestimated to have the same network latency with the server, or otherparticipants in a group call, and so forth. In one example, optionalstep 440 may be performed when there is a change in the type of packetloss protection that is selected, e.g., due to a changing networklatency estimate crossing the threshold as compared to a previousestimate.

At optional step 450, the processing system may apply, to thenetwork-based communication, the type of packet loss protection that isselected. For example, the processing system may include one or bothendpoints (or multiple endpoints) of the network-based communication.Thus, for a FEC packet loss protection scheme, the processing system maytransmit FEC packets and/or may be configured to receive FEC packets andto recreate any lost packets in accordance with information in the FECpackets, and so forth. Similarly, for a retransmission-based scheme, theprocessing system may transmit packets without FEC and/or may beconfigured to receive packets without FEC, to request retransmission forlost packets, etc.

Following step 430, or one of the optional steps 440 or 450, the method400 proceeds to step 495 where the method ends.

It should be noted that the method 400 may be expanded to includeadditional steps, or may be modified to replace steps with differentsteps, to combine steps, to omit steps, to perform steps in a differentorder, and so forth. For instance, in one example the processor mayrepeat one or more steps of the method 400, such as steps 410-430, steps410-440, etc., such as for additional estimates of network latency forfuture time periods, and so on. In another example, the method 400 mayinclude selecting the level of packet loss protection for when FEC isselected. For instance, FEC can be set to cover 4% packet loss, 5%packet loss, etc., where up to 4% of packets, 5% of packets, etc. can belost and the receiver is still able to recover all packets. If there isgreater packet loss, some packets may not be recoverable. However, ifthere is less packet loss than the level that is selected, some FECpackets are extraneous and may result in bandwidth waste. In anotherexample, the method 400 may include selecting the threshold. Forinstance, the threshold may be different for different applications,which may have different delay/latency tolerances. In still anotherexample, the method 400 may be expanded to include operations of themethod 500 below, e.g., for network-based communications comprisingimmersive visual streams. Thus, these and other modifications are allcontemplated within the scope of the present disclosure.

FIG. 5 illustrates a flowchart of a method 500 for selecting a type oftransmission for an immersive visual stream based upon a latencyestimate, in accordance with the present disclosure. In one example, themethod 500 is performed by a component of the network 100 of FIG. 1,such as by one of application servers 114, servers 149, mobile devices157A, 157B, 159, 167A, 167B, or 169, home network devices, such as STBs162A and 162B, TV 163A and TV 163B, home phone 164, PC 166, and soforth, and/or any one or more components thereof (e.g., a processor, orprocessors, performing operations stored in and loaded from a memory),or by a plurality of such devices in conjunction with one another, andso on. Similarly, the method 500 may be performed by a component of thesystem 200 of FIG. 2, such as client device 210A, client device 210B,server 240, latency guidance server 250A, or latency guidance server250B, and/or any one or more components thereof (e.g., a processor, orprocessors, performing operations stored in and loaded from a memory),or by a plurality of such devices in conjunction with one another, andso on. In one example, the steps, functions, or operations of method 500may be performed by a computing device or system 600, and/or processor602 as described in connection with FIG. 6 below. For instance, thecomputing device or system 600 may represent any one or more componentsof one or more components of the network 100 of FIG. 1 and/or the system200 of FIG. 2 that is/are configured to perform the steps, functionsand/or operations of the method 500. Similarly, in one example, thesteps, functions, or operations of method 500 may be performed by aprocessing system comprising one or more computing devices collectivelyconfigured to perform various steps, functions, and/or operations of themethod 500. For instance, multiple instances of the computing device orprocessing system 600 may collectively function as a processing system.For illustrative purposes, the method 500 is described in greater detailbelow in connection with an example performed by a processing system.The method 500 begins in step 505 and proceeds to step 510.

At step 510, the processing system obtains a latency estimate for animmersive visual stream. For instance, the immersive visual stream maycomprise data for rendering at least one of: 3D visual content (e.g.,including video and non-video, such as exploratory 3D models ofbuilding, environments, ships, aircraft, biological systems, etc.),virtual reality content, augmented reality content, or volumetric videocontent.

At step 520, the processing system determines whether the latencyestimate exceeds a latency threshold for selecting a type oftransmission for the immersive visual stream. The immersive visualstream may be delay/latency sensitive, such as for a video call, gaming,live streaming, etc. For instance, the latency threshold may be 20 msfor 4K video streams, 40 ms for visual qualities less than 4K, 100 msfor animated visual streams, etc.

At step 530, the processing system selects the type of transmission forthe immersive visual stream from among a field of view restricted typeof transmission and a field of view plus out of scene type oftransmission based upon the determining. For instance, when the latencyestimate is determined to not exceed the latency threshold, the field ofview restricted type of transmission is selected, and wherein when thelatency estimate is determined to exceed the threshold, the field ofview plus out of scene type of transmission is selected. In one example,the field of view restricted type of transmission includes visualinformation for regions within the field of view of a user and does notinclude visual information for regions outside of the field of view ofthe user. In addition, in one example, the field of view plus out ofscene type of transmission includes first visual information for regionswithin the field of view of a user and second visual information forregions outside of the field of view of the user. For instance, thefirst visual information may comprise a higher visual quality than thesecond visual information, e.g., from a higher quality track of the samevideo or other visual content. In another example, the first visualinformation may comprise at least one additional layer of a layeredvideo as compared to a number of layers for the second visualinformation.

At optional step 540, the processing system may send an instruction toapply to the immersive visual stream the type of transmission for theimmersive visual stream that is selected. For instance, the processingsystem may comprise a centralized and/or network-based controllersending instructions to a client or server of the immersive visualstream (or to two participant device/endpoints for applications withinteractive visual streams, such as 3D remote collaboration, gaming,etc.). In another example, the processing system may comprise aparticipant device (e.g., a recipient device) sending the instruction toanother participant device (e.g., a server/transmitter) to apply thetype of transmission. In one example, optional step 540 may be performedwhen there is a change in the type of transmission for the immersivevisual stream that is selected, e.g., due to a changing network latencyestimate crossing the threshold as compared to a previous estimate.

At optional step 550, the processing system may apply, to the immersivevisual stream, the type of transmission for the immersive visual streamthat is selected. For example, the processing system may comprise atransmitting participant device that applies the type of transmissionthat is selected at step 530.

Following step 530, or one of the optional steps 540 or 550, the method500 proceeds to step 595 where the method ends.

It should be noted that the method 500 may be expanded to includeadditional steps, or may be modified to replace steps with differentsteps, to combine steps, to omit steps, to perform steps in a differentorder, and so forth. For instance, in one example the processor mayrepeat one or more steps of the method 500, such as steps 510-530, steps510-540, etc. In another example, the method 500 may include selectingthe latency threshold. For instance, the threshold may be different fordifferent applications, which may have different delay/latencytolerances. For example, certain applications may use a longer buffer,such a “live” video streaming, which may actually be transmitted and/orreceived with a several second delay, or the like, whereas video callsmay use a shallow buffer or no buffer. In still another example, themethod 500 may be expanded to include operations of the method 400above, e.g., to include both packet loss protection adaptation and typeof transmission adaptation based upon latency guidance. Thus, these andother modifications are all contemplated within the scope of the presentdisclosure.

In addition, although not expressly specified above, one or more stepsof the method 400 or the method 500 may include a storing, displayingand/or outputting step as required for a particular application. Inother words, any data, records, fields, and/or intermediate resultsdiscussed in the method can be stored, displayed and/or outputted toanother device as required for a particular application. Furthermore,operations, steps, or blocks in FIG. 4 or FIG. 5 that recite adetermining operation or involve a decision do not necessarily requirethat both branches of the determining operation be practiced. In otherwords, one of the branches of the determining operation can be deemed asan optional step. Furthermore, operations, steps or blocks of the abovedescribed method(s) can be combined, separated, and/or performed in adifferent order from that described above, without departing from theexample embodiments of the present disclosure.

FIG. 6 depicts a high-level block diagram of a computing device orprocessing system specifically programmed to perform the functionsdescribed herein. For example, any one or more components or devicesillustrated in FIG. 1 or FIG. 2, or described in connection with themethod 400 of FIG. 4 or the method 500 of FIG. 5 may be implemented asthe processing system 600. As depicted in FIG. 6, the processing system600 comprises one or more hardware processor elements 602 (e.g., amicroprocessor, a central processing unit (CPU) and the like), a memory604, (e.g., random access memory (RAM), read only memory (ROM), a diskdrive, an optical drive, a magnetic drive, and/or a Universal Serial Bus(USB) drive), a module 605 for selecting a type of packet lossprotection for a network-based communication based upon a latencyestimate and/or for selecting a type of transmission for an immersivevisual stream based upon a latency estimate, and various input/outputdevices 606, e.g., a camera, a video camera, storage devices, includingbut not limited to, a tape drive, a floppy drive, a hard disk drive or acompact disk drive, a receiver, a transmitter, a speaker, a display, aspeech synthesizer, an output port, and a user input device (such as akeyboard, a keypad, a mouse, and the like).

Although only one processor element is shown, it should be noted thatthe computing device may employ a plurality of processor elements.Furthermore, although only one computing device is shown in the Figure,if the method(s) as discussed above is implemented in a distributed orparallel manner for a particular illustrative example, i.e., the stepsof the above method(s) or the entire method(s) are implemented acrossmultiple or parallel computing devices, e.g., a processing system, thenthe computing device of this Figure is intended to represent each ofthose multiple general-purpose computers. Furthermore, one or morehardware processors can be utilized in supporting a virtualized orshared computing environment. The virtualized computing environment maysupport one or more virtual machines representing computers, servers, orother computing devices. In such virtualized virtual machines, hardwarecomponents such as hardware processors and computer-readable storagedevices may be virtualized or logically represented. The hardwareprocessor 602 can also be configured or programmed to cause otherdevices to perform one or more operations as discussed above. In otherwords, the hardware processor 602 may serve the function of a centralcontroller directing other devices to perform the one or more operationsas discussed above.

It should be noted that the present disclosure can be implemented insoftware and/or in a combination of software and hardware, e.g., usingapplication specific integrated circuits (ASIC), a programmable logicarray (PLA), including a field-programmable gate array (FPGA), or astate machine deployed on a hardware device, a computing device, or anyother hardware equivalents, e.g., computer readable instructionspertaining to the method(s) discussed above can be used to configure ahardware processor to perform the steps, functions and/or operations ofthe above disclosed method(s). In one example, instructions and data forthe present module or process 605 for selecting a type of packet lossprotection for a network-based communication based upon a latencyestimate and/or for selecting a type of transmission for an immersivevisual stream based upon a latency estimate (e.g., a software programcomprising computer-executable instructions) can be loaded into memory604 and executed by hardware processor element 602 to implement thesteps, functions or operations as discussed above in connection with theexample method(s). Furthermore, when a hardware processor executesinstructions to perform “operations,” this could include the hardwareprocessor performing the operations directly and/or facilitating,directing, or cooperating with another hardware device or component(e.g., a co-processor and the like) to perform the operations.

The processor executing the computer readable or software instructionsrelating to the above described method(s) can be perceived as aprogrammed processor or a specialized processor. As such, the presentmodule 605 for selecting a type of packet loss protection for anetwork-based communication based upon a latency estimate and/or forselecting a type of transmission for an immersive visual stream basedupon a latency estimate (including associated data structures) of thepresent disclosure can be stored on a tangible or physical (broadlynon-transitory) computer-readable storage device or medium, e.g.,volatile memory, non-volatile memory, ROM memory, RAM memory, magneticor optical drive, device or diskette and the like. Furthermore, a“tangible” computer-readable storage device or medium comprises aphysical device, a hardware device, or a device that is discernible bythe touch. More specifically, the computer-readable storage device maycomprise any physical devices that provide the ability to storeinformation such as data and/or instructions to be accessed by aprocessor or a computing device such as a computer or an applicationserver.

While various embodiments have been described above, it should beunderstood that they have been presented by way of example only, and notlimitation. Thus, the breadth and scope of a preferred embodiment shouldnot be limited by any of the above-described example embodiments, butshould be defined only in accordance with the following claims and theirequivalents.

What is claimed is:
 1. A method comprising: obtaining, by a processingsystem including at least one processor, a latency estimate for animmersive visual stream; determining, by the processing system, whetherthe latency estimate exceeds a latency threshold for selecting a type oftransmission for the immersive visual stream; and selecting, by theprocessing system, the type of transmission for the immersive visualstream from among: a first type of transmission that comprises onlyvisual data for a predicted field of view area and a second type oftransmission that comprises visual data for an actual field of viewarea, based upon the determining, wherein when the latency estimate isdetermined to not exceed the latency threshold, the first type oftransmission is selected, and wherein when the latency estimate isdetermined to exceed the latency threshold, the second type oftransmission is selected.
 2. The method of claim 1, wherein the firsttype of transmission includes visual information for regions within thepredicted field of view area of a user and does not include visualinformation for regions outside of the predicted field of view area ofthe user.
 3. The method of claim 2, wherein the second type oftransmission includes first visual information for the regions withinthe actual field of view area of the user and second visual informationfor the regions outside of the actual field of view area of the user. 4.The method of claim 3, wherein the first visual information comprises ahigher visual quality than the second visual information.
 5. The methodof claim 4, wherein the first visual information comprises at least oneadditional layer of a layered video as compared to a number of layersfor the second visual information.
 6. The method of claim 1, furthercomprising: sending an instruction to apply to the immersive visualstream the type of transmission for the immersive visual stream that isselected.
 7. The method of claim 1, further comprising: applying, to theimmersive visual stream, the type of transmission for the immersivevisual stream that is selected.
 8. The method of claim 1, wherein theimmersive visual stream comprises data for rendering at least one of:three-dimensional visual content; virtual reality content; augmentedreality content; or volumetric video content.
 9. The method of claim 1,wherein the processing system comprises at least one of: a recipientdevice of the immersive visual stream; or a transmitting device of theimmersive visual stream.
 10. The method of claim 1, wherein theprocessing system comprises a network-based processing system.
 11. Adevice comprising: a processor; and a computer-readable medium storinginstructions which, when executed by the processor, cause the processorto perform operations, the operations comprising: obtaining a latencyestimate for an immersive visual stream; determining whether the latencyestimate exceeds a latency threshold for selecting a type oftransmission for the immersive visual stream; and selecting the type oftransmission for the immersive visual stream from among: a first type oftransmission that comprises only visual data for a predicted field ofview area and a second type of transmission that comprises visual datafor an actual field of view area, based upon the determining, whereinwhen the latency estimate is determined to not exceed the latencythreshold, the first type of transmission is selected, and wherein whenthe latency estimate is determined to exceed the latency threshold, thesecond type of transmission is selected.
 12. A non-transitorycomputer-readable medium storing instructions which, when executed by aprocessing system including at least one processor, cause the processingsystem to perform operations, the operations comprising: obtaining alatency estimate for an immersive visual stream; determining whether thelatency estimate exceeds a latency threshold for selecting a type oftransmission for the immersive visual stream; and selecting the type oftransmission for the immersive visual stream from among: a first type oftransmission that comprises only visual data for a predicted field ofview area and a second type of transmission that comprises visual datafor an actual field of view area, based upon the determining, whereinwhen the latency estimate is determined to not exceed the latencythreshold, the first type of transmission is selected, and wherein whenthe latency estimate is determined to exceed the latency threshold, thesecond type of transmission is selected.
 13. The non-transitorycomputer-readable medium of claim 12, wherein the first type oftransmission includes visual information for regions within thepredicted field of view area of a user and does not include visualinformation for regions outside of the predicted field of view area ofthe user.
 14. The non-transitory computer-readable medium of claim 13,wherein the second type of transmission includes first visualinformation for the regions within the actual field of view area of theuser and second visual information for the regions outside of the actualfield of view area of the user.
 15. The non-transitory computer-readablemedium of claim 14, wherein the first visual information comprises ahigher visual quality than the second visual information.
 16. Thenon-transitory computer-readable medium of claim 15, wherein the firstvisual information comprises at least one additional layer of a layeredvideo as compared to a number of layers for the second visualinformation.
 17. The non-transitory computer-readable medium of claim12, wherein the operations further comprise: sending an instruction toapply to the immersive visual stream the type of transmission for theimmersive visual stream that is selected.
 18. The non-transitorycomputer-readable medium of claim 12, wherein the operations furthercomprise: applying, to the immersive visual stream, the type oftransmission for the immersive visual stream that is selected.
 19. Thenon-transitory computer-readable medium of claim 12, wherein theimmersive visual stream comprises data for rendering at least one of:three-dimensional visual content; virtual reality content; augmentedreality content; or volumetric video content.
 20. The non-transitorycomputer-readable medium of claim 12, wherein the processing systemcomprises at least one of: a network-based processing system; arecipient device of the immersive visual stream; or a transmittingdevice of the immersive visual stream.